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Ibstb&ct 

^l^r^r-^- / An empirical study was conducted in order to obtain 

«f0ment effect estimates with the Special Regression .model for 
Pidup's in which therp was no treat lent. General matheTmatics test 
[l^fefis nete obtained! trom 730 ninth graders in .city schbtei^^ somewhat 
Jimilir to Title I schools, but in which no special treatm^Ats were 
'""veil. Hypothetical experimental and control g-roups were formed^ on 

e- basife of pretest scoreis and the data were analyzed acqor ding <|o 
JSie BMC Model C (special regression model). Conclusions were that: 
fey eVem minor ceiling or floor effects resulted in nonhombgeneous 
llfi^esslon lines for treatment and compit^son groups; <2) tests of 
piat^stlcal sigiiif icance shou3,d precee^ conclusions of educational 
§liiiniflcance at the local school or district level; and (3) 
feiaelines to assist with interpreting Evaluation results are needed. 
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A be tract 



An empirical study was conducted In order to ojbtaln treatment effect ' 
estimates with the Speojial Regression mo<lel for groups In which there was no 
4(i^eatment« Results for the Special Regression Title J evaluation li^el were 
calculated for approximately 730 ninth graders enrolled In general math classes In 
a large city school system. Conduslons were that^: (1) even minor celUn^r floor 
effects resulted In nonhomogenous regression llqes for treatnient and comparison 

! I ■ , . " ■ 

groups^ (2) tests of statistical significance should preceed conclusions of educational 
significance at the local school or district leveU and (3) guidelines to assist with 
Interpreting evaluation results ^re needed. 



.0J)8ftrved treatmen^.^f£^ct8 with Special 
J^egreaalon Evailuatloh Models' ln\tSloups 



iv/ . ^ with No Treatment 



Gary D. Estes •. . 

S* ' Northwest Regional Educj^tlonal Laboratory " ' ~ ■ ' i' 

; Judith I. Anderson - 

The BDM Cor ^ration ' - * . 

• . . » • * . • . 

f * ~ • " ._ ' A . 

The Elementary and Secondary fiducatlon Act of 1966 initiated federal aid to 
education on a large scale. The amount of ESEA Title I funds has steadily Increased 
Sl'iwe J.965 and concurrently the number and type of compensatory programs^ supported 
with ESEA Title I funds have also Increased. Evaluations of these projec^» have ofkaA' 

- • ' ' ' ' ' ■■• 

been poorly designed and implemented; even when this has not been the case, comparison 
across projects has been difficult due to the wide variety of evaluation designs and 
testing instruments employed. In an effort to assess the impact of ^Itle Iprogranis 
nationwide In a consistent and meaningful manner, thi| UfS. Office of Education has . 
provided districts with three basic evaluation designs. / 

The impact of a Title I project is defined by the. three evaluation designs as the 
difference betwcf^n the Title I students' observed performance and an estimate of 
what their performance vi-ould have been in th^ absence of the Title I program. The 
thre^'fevaluation designs' provided are: Model a/ the Norm-Referenced Model; ' • 

Model B, the Control Group Model; and Model C, the Speclat Regress lop Model. 
This study will provide information on the Special Regression evaluation model. 
The specif ip objectives are (1) to estimate treatment effect when the Special Regression 
model Is used with a group in which there was no treatment,; i.e., does the Special 
Regression model provide an estim.ate of zero treatment effects when applied to actual 
data in which there was no treatment ? , and (2) to provide recommendations which 



mtsht be Viaed In Interpreting the results obtained wlthjthe S|peoIaI Ai^resslon 
1 ^ model In- particular and all models ,ln general* * 

■ . V " . ^ . ' : . •; 

The Special Regression model actually makes use of two regression modelsy the 
Itjdgre894on Projection model and the Regression Discontinuity mod6U The designs \ * 
are useful when alt students below a designated cut-off score are assigned to the 'Title I 

- . . • • • ' ' * - 

program, i.e. , the treatment group, and lio.^student at or^above the'cut-^off score la 

• - ^ • ' - * * " ) ■ 

assigned to the Title I program; L e, , the comparison gr^up. A strefngth of the Special 



Regression model Is that It provides a sounder basis for establishing no-treatm^t 
posttest e^ectatlons than national nofms (Hotsty Tallmadge & Wood, 1975). A weakness 
of the Special Regression model Is th'b assumption that a single regression line will fit^ X 

' ^^'V . • ; -■ ■ 

the treatment and comparison groups under no-treatment conditions. The Regression 

Projection model also, Involves the assumption tbat a linear relationship exists between 

'"''^ . . - ^1 ^ f 

the pretest and po^test under no-treatment conditions. However, the problem of 

curvilinear relationships can be eliminated Iby using higher order regression equations 

(Horst, «t al, 1975), An additional problepi^is that there may be no difference in 

regression line Intercepts when lowest scoring Title I students make the largest 

gains — thereby masking a significant overall treatment effect. Proce'clures outlined ^ 

by Ward and Jennings <1973, p. 128) can be used to process data In which there might 

V be an Interaction present. 



Method 



tiata Source. Approximately 730 ninth grade students enrolled in genJral 



mathfsmatlcs classes In schoolis somewhat similar to the District's Title I schools 



J '1 



s.erved as subj^is In this^tudy. Three tests of basic mathematical skills were 
administered to the students in October and May.* The three tests are described below« 



ERIC 



. The Comprehensive Tests (tf Basic SkUts.(CTBS)Math Computation subtest ' 
which measures basic arittimetrc skills. Form S, Level 4 was given. The test , 
contains 48 multiple choice Items and takes 40 minutes to administer. - 

The Shaw-Hlehle^Ijdividuaflzed Computational' Skills Te^t is 'a 60 Item untlmed* 
test coverlhg^ddltCon, subtraction, multiplication and dlvlsioh of Whole numbers, ; 
fi^acttons and decimals, as' well as word p^roblems. The test is ndt multiple choice; 
that Is,, students must write In the correct answers. 

The Minimal Mathematlcs^Pxoflclency Test (MMPT) was devcrDoped by the school 



•district and consists of 64 multiple choice items covering basic arithmetic skills. 

.-i^— -7" * - — . " ' . 

The items are grouped Into Ifi areas with four items per area» and the student 
receives one^ppint for each area passed. Once a student passes an area by correctly 
answering at least three of the four Items In that area, he or she is not tested with 

\. ' ■ . ■■ ■ ■ ' - 

those Items again, ue« , at posttestlng'the student receives only^enls in those"^ 

' ' ■ ■ i ' 

' areas missed on the pretest. 

• ' . * ■ •* 

Of the 730 students, 316. had pre-and posttest scores on the CTBS.Math 

' ^ • ' " ■ i , - ^ - ... 

Computations subtest, 229 had pre-and posttest scores in the S^w-Hiehle, and 473 

l^ad pre- and posttest scores on the MMPT. Cut-off scores which were comparable 

to those used to assign students to ESEA Title I math classes were Identlfleid for the 

three tests; these are Included In Tables 1-3. Stiidenttii^ with pretest scores at oi* 

below the cut-off scores were labeled the 'treatment group" and students with 

pretest scores above the cut-off scores serve'Jl as the "coii^arison grotp". 

Analyses. Procedures outlined by Tallmadge and Wood (1976) were used to 

calculate program Impact. Both the Regression Projection taodel, wher^ treatment 

effect is taken to be the differe^e between the actual and estimated mean scores and 



Table 1 • * 



Summary'jof Retsulta on, the CTBS Math 



Treatiment 



Group 



Control 



Number with Pre- an< 
Posttest Scores 

PreWst Mean 
S. D. 

.Posttest Mean 
S. D. 



Pre-Post Correlation 
Slope 

Cut-off Score 

Combined Groups Pre-Post 
•Correlation 



130 . 



416,0 
34.4 



471.9 
61.1 

.26 
.46 



457 



.62 



.1 



186 



521.6 
39. 8 

541.6 
64.5 

.55 
.89 



Table 2 ' 
Summary of Results on the ShaW'^Hiehle 



Group 



Treatment 



Control. 



Number, with Pre-* and 
Posttest Scores 

Pi^test"^ Mean 
■S. D. 

■ • ■ 

Posttest I Mean 
S. D. 

Pri^-Post Correlation 
Slope 

Cuf •'Off Score 

Combined Groups Pre-Pgst 
Correlation 



111 



22.6 
5.78 

29.0 
7.49 

.540 
.700 



30 



.70 



118 



38.4 
5.^2 

39.2 
7.98 

.506 
.774 



Table 3 

Summaify^of Resultls on the MMPT 

' I I I I , I 





Treatment 




Control 


4 

jNuxnoer wun ^ve^ ana 
Posttest Scores ' 




r 

• 

V 


368 

/; : ; 


• « 
Pretest Mean 


• 

3.44 
1,29 


* 


10.96 
. ; 3.20 


■ • .■ k 
/ S. D, 


in IS 
3. 86 




1,61 


. Pre-Post Correlsitlon 
Slopei 1 
Cut-ojff Score ^ 
Combined Groups Pre- Post 
Correlation 


.260 
.777 

# 


« 

5 ; 

.66 


.453 
.228 


» . I" 


Table 4 






^ Summary of Hesults of Stgnif Icance Tests 






Type of Analysis 




Measurement Reez)esslon Projection 
Instrument NCE G^in t 

at Mean 


ReKressIbn ]biscontinuity 
NGEG^ln t 
atC3ut-8ff/ « 


CTBS 4.40 


3.14* 


■ ^./ 
i.ao 


0.75 


Shaw-HIehle 2.15 

• 


1.78* 


2.90 


6.57* 


MMPt -6.30 


5.95* 


. -4.42 


6.22* 



"^Significant at the • 05 leveL 



the Regreirflbn DlBOonttaulty model, where the difference between the treatment 



groap and the comparison group r egress Ion lines Is tested at the point 



where they 



Intersect the pretest- cut-off secret were used. Results were presented In Normal 
Xurve Equivalents. In Addition, the significance tests propq^ed by Horst, Tallmadg^ & 
Wod<l,([197S) were calculated, (See Table 4). , 

Results tind Conclusions 

Estimates of treatment Intact at the pretest mean and cut-off score* 

respectively, were 4.4 and 1.3 Normal Curve Equivalents (NCE's)for the CTBS, 

♦ ■ 

-6.30 and -4.42 NCE's for the MMPT, and 2.15 and 2.90 NCE'a for the Shaw-Hlehle. 
AUjt tests used to compare results for the ''treatment" and "comparison" groups 
werei significant at the . 05 level ekcept for regression discontinuity analysis on 
the CTBS. 

.A slight floor .effect was observed for the "treatment gmqp" on the CTBS 



pretest and a notable celling effect was observed l^or the ^'comparison group" on 
the MMPT .posttest. No cellli^ or floor effects were ob|3erved on the Shaw-Hlehle. 
Pretest-posttest correlations were moderate: CTBS, .62; Shaw-Hlehle, . 70; and 
MMPT, .66. 

On the CTBS and the MMPT, results were consistent with what would have been 
e^qpected given the observed celling and floor effects, l.e; diffeirences In treatment 

... • . .:. ' I 

^effect estimates at the pretest mean and cut-off score. However, on the Shaw-Hlehle, 
where no celling or floor effects were encountered, comparable positive gains were 
found using fhe regression projection and the regresstoh discontinuity ^proaches. 
Tallmadge and Wood (1976, p. 61) state that, ' 



'*ir both.m«iittr«t of the trMtment effect ihow positive galne ol 
oomper Able magnitude! , .toer e can be; little doubt that the . ^ ■ • 
treatment did, In faot, have libpapt**'' ' ' • 

fThe propoeed prooedures do not recommend examining resulta for statlstloali 
algnlflcanoe. Slgnlfloanoe. testa outlined by Horst et ftl (1975) were employed, ' 
however, and would lead one to cofnolude that,- based on pezfairm^anOe^<^^^^ 
ShawHlehle and the MMPT, a significant treatment effect existed, although In 
opposite directions, 1. e. , the effe6t was positive In the Shaw-Hlehle and negative 
on the MMPT. Ttie significant negative treatment effects observed on the MMPT . 
can eas^lly be explained by celling effects and to a lesser extent the CTBS results , 
might be a result of floor effects If one examines the slopes In Tables. 1-3. Thus, 

• "'■ ■■-.„^^ ■ ■ \- ■ , . . ■; ■ . 

these results suggest that It Is possible that even minor celling, or floor ejects 
might J^ve.slgnmcantelfeot on ».eMlng project impact; / 

It Is recommended that If local a choql districts are to meet the objective stated 
by Tallmadge and A^^po^ (1976) **to provide meaningful, cony^arable Information about 



J|*ltle Iprojects*-\ that 



close attentlcm be paid, to model assumptions such as floor/ 



celling effects and coi]^blned grbiq>*s linearity In the absence of treatment. Fttriher,- 
It is recommended that additional study be made of the consequences ojt violating 



/ 



these assuniptions In order to ensureNthat, In fact, there Is a treatment tn effect. 
In cases where celling and floor effects, are observed. It might be advisable to 
consider Alternatives to Including all treatment or comparison students in the analys^s*^ 
Specifically, if a ceiling effect Is noted for the coix^artson group, an analysis whtch 



deleted all students scortng above^a *'cell&ig" cut-off. score might .J)rpvtde results 
which avoided celling effects^and allowed ioi stronger Interpretations of results. 
In fact, it is possible. that the procedure of Incltding'atl st^dje&ts who do act receive " ; 
the;^it1e I treatment in the comparison group might be modified to include only a 



■ufflbitnt number and^.range of itudenti to provtde'an aeour^^e estimate of the 



Q which aUowod atudoilft- 



domparlaon group, regreieion line. Testing* Title I and all obmparlioa atudentt with 
a single test and avoiding floor effects with title I students afxl oelllni eifeots within -n 
the odmpar ison 8>oup might prove to be dtff louft. A .prdo 

with celling effects to be deleted from the evaluation analysis {would protjably assist , 
■ to ffiroldvl^e results produced on the MMPT In Table 4. 

It Is generally recommended that prei^poettbst correlation^ be sufficiently high In 
order to compute a special regression analysla." The basis lor| this recommendation ' 

■ / ■ \ ■ • ■ '/'^ :' ' ' ■ 

la to Insure some degree, of stfibllity In the estimates made hrom the riegresslon 
analyses. The restriction In ^ange within groups which la produced when selection 
for group membersh^ Is based on the pretest score will automatically lower the' * 
pre-posttest corrections. Although the.pxe-posttest correlatl<»i8 for the three • 
^ criteria used In this study would likely be judged adequate; It Is possible that' the 

pre-posttest cQrr^ations ranglijig;.from . 26 to ^ 55 'within the treatment and ^iitrol ' 

f ' ■ ' ■ ' ' ■ ' ■ ■ 

groups would have been judged Inadequate. If' procedures outlined by Guilford 

{1^65, p. 341-344), .were used to adjust these correlations for resttlctlbn In range," 

' I ■ ' . ■ .-. ; - \ 

the adjusted car relations cOuld then be examined for adequacy. Thus,. It is recommended'' 

that ^^Ithinrgroup correlative in the special regression evaluation models be Inte'irpret^ 

. jrfter restriction in range Is taken Into cons IdeJatloa, , ' '/ \ 

- ' "■' ' '■ 

The sea!9}le sizes in this study were reasohably large and statistically significant 

• '- ■■ * " ■ . ■ ■ ■ , ' . ■ X ■ • ■■ 

gains of two to three KCE's were observed on the Shaw-Hlehle for which there w^r« ' 
no floor br^ celling effectsi Some guidelines for Interpreting evahihtlon results in ■ ,< 
addition to those air eadybrovlded are needed. These guidelines might Include /^^ 
l_factors dwjh as gains observed in other projects and,-objectLve8 based on estimated / 1> 



acjdevei^ent increases as a result of Title I services, e.g., a ten percent Increase 

>•'■"•••' , ■ . I' 



lni aohltvem«Bt. | It !■ potalble that the dlBtributlona of NpE^galM .will be uiaftil tn 
, eatabllihliig "meaDlngfur* project objeotlves as data are reported under the prqpoiiad- 
Title I evaluation models. In any events , we would recommend against Interpriktlng 

.-■ , ^ , . . • .a 

/ . • : . - ■ ' ' 

"MX NCE gains 4fl poelttve evidence of program fluocess". ' 

Finally, It^Rraas^ that users of the Special RegresslQn Title I eyaluallc|b\ model 
wlll'need to be particularly sensitive to restrlotlons of model aasumptloiui. ]t>v^B ^ ' 
concluded that the regi^esston e^valuatlQn might be' more iq;>plioable if procedures 

which deleted data oausidg floQ^ or cejillng elfbcts Were adopted. Based upon the ^ 

. . i, . ■ •' ■ • ' ^ • -•, , ■ 

results of the Shaw-Hlehle Wst, It appears' that guidelines in addition to tc^sts of 

Statistical significance, and comparable Impact estimates at the out-ott mod pretest 

means will be needed In order to ftiake meaningful interpretattpns of ewluatlcn result^. 

Thc\comparl8dn of gains at the cut-off point and pretest means Is a c/iWl? method^ to 

assist ^^n Judging whether the treatment and comparison group regre^elon lines are 

homogeneot^« Lit ml^t be reasonal^e to use actual tests ^ slgnlflcj/anoe In assessing 

the homogeneity of regression linese . / ' , . 

. . ... ; ■ . , ■ . _ / ' ■ - / 

In concluston» the data In this paper provided evidence that care Is needed In 

• • ' ---^ , - , . "» 

- I' » ' ' - 

conductik:^ and Intrepretlng evaluations using the Special R^ress Ion model pressed 

- ' . .' 'V • f / 

for Title I program evali^atione* The potential usefulness of the mddel wlU be realized - /i 
dnly if procedures and techniqtles are used with the model which assist to iniure t^at- 
the infQrmatlon«contained in the data Is teased out and not lost or misinterpr^ed. ^ / / 
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